Development and Validation of a Scoring Rubric for Editorial Evaluation of Peer-review Quality: A Pilot Study.

Introduction: Despite the importance of peer review to publications, there is no generally accepted approach for editorial evaluation of a peer review's value to a journal editor's decision-making. The graduate medical education editors of the Western Journal of Emergency Medicine Special Issue in Educational Research & Practice (Special Issue) developed and studied the holistic editor's scoring rubric (HESR) with the objective of assessing the quality of a review and an emphasis on the degree to which it informs a holistic appreciation for the submission under consideration. Methods: Using peer-review guidelines from several journals, the Special Issue's editors formulated the rubric as descriptions of peer reviews of varying degree of quality from the ideal to the unacceptable. Once a review was assessed by each editor using the rubric, the score was submitted to a third party for blinding purposes. We compared the performance of the new rubric to a previously used semantic differential scale instrument. Kane's validity framework guided the evaluation of the new scoring rubric around three basic assumptions: improved distribution of scores; relative consistency rather than absolute inter-rater reliability across editors; and statistical evidence that editors valued peer reviews that contributed most to their decision-making. Results: Ninety peer reviews were the subject of this study, all were assessed by two editors. Compared to the highly skewed distribution of the prior rating scale, the distribution of the new scoring rubric was bell shaped and demonstrated full use of the rubric scale. Absolute agreement between editors was low to moderate, while relative consistency between editor's rubric ratings was high. Finally, we showed that recommendations of higher rated peer reviews were more likely to concur with the editor's formal decision. Conclusion: Early evidence regarding the HESR supports the use of this instrument in determining the quality of peer reviews as well as its relative importance in informing editorial decision-making.

A Collaborative Approach to Mentored Peer Reviews Sponsored by the Council of Residency Directors in Emergency Medicine.

Introduction: Historically, there have been no systematic programs for teaching peer review, leaving trainees to learn by trial and error. Recently, a number of publications have advocated for programs where experienced reviewers mentor trainees to more efficiently acquire this knowledge. Objective: Our goal was to develop an introductory learning experience that intentionally fosters peer-review skills. Methods: The Council of Residency Directors in Emergency Medicine (CORD) offered education fellowship directors the opportunity to mentor their fellows by reviewing submitted manuscript(s) supplemented by educational material provided by their journal. Reviews were collaboratively created. The decision letter that was sent to manuscript authors was also sent to the mentees; it included all reviewers' and editor's comments, as feedback. In 2022, fellows received a post-experience survey regarding prior experiences and their perspectives of the mentored peer-review experience. Results: From 2020-2022, participation grew from 14 to 30 education fellowships, providing 76 manuscript peer reviews. The 2022 survey-response rate of 87% (20/23) revealed that fellows were inexperienced in education scholarship prior to participation: 30% had authored an education paper, and 10% had performed peer review of an education manuscript. Overall, participants were enthusiastic about the program and anxious to participate the following year. In addition, participants identified a number of benefits of the mentored experience including improved understanding of the scholarship process; informing fellows' scholarly pursuits; improved conceptualization of concepts learned elsewhere in training; and learning through exposure to scholarship. Conclusion: This program's early findings suggest that collaboration between academic societies and interested graduate medical education faculty has the potential to formalize the process of learning peer review, benefitting all involved stakeholders.

SBAR-LA: SBAR Brief Assessment Rubric for Learner Assessment.

Introduction: Structured communication tools are associated with improvement in information transfer and lead to improved patient safety. Situation, Background, Assessment, Recommendation (SBAR) is one such tool. Because there is a paucity of instruments to measure SBAR effectiveness, we developed and validated an assessment tool for use with prepractice health professions students. Methods: We developed the SBAR Brief Assessment Rubric for Learner Assessment (SBAR-LA) by starting with a preliminary list of items based on the SBAR framework. During an interprofessional team training event, students were trained in the use of SBAR. Subsequently, they were assigned to perform a simulated communication scenario demonstrating use of SBAR principles. We used 10 videos from these scenarios to refine the items and scales over two rounds. Finally, we applied the instrument on another subset of 10 students to conduct rater calibration and measure interrater reliability. Results: We used a total of 20 out of 225 videos of student performance to create the 10-item instrument. Interrater reliability was .672, and for eight items, the Fleiss' kappa was considered good or fair. Discussion: We developed a scoring rubric for teaching SBAR communication that met criteria for validity and demonstrated adequate interrater reliability. Our development process provided evidence of validity for the content, construct, and response process used. Additional evidence from the use of SBAR-LA in settings where communication skills can be directly observed, such as simulation and clinical environments, may further enhance the instrument's accuracy. The SBAR-LA is a valid and reliable instrument to assess student performance.

"Disbelief and Sadness": First-Year Health Profession Students' Perspectives on Medical Errors.

OBJECTIVES: The aims of the study were to examine the reactions of first-year health profession students to medical errors and determine whether differences exist between disciplines. METHODS: After viewing the Team STEPPS Program's Susan Sheridan video that describes two separate medical errors, students from anesthesia assistant, medical imaging, medicine, nursing, physical therapy, and physician assistant programs provided unstructured open responses reflecting on initial impressions of medical errors depicted in the film. Student responses were assessed via inductive coding techniques and thematic analysis and stratified by discipline. Frequencies of key themes were calculated. Descriptive analyses characterized respondents and χ2 tests compared responses between disciplines. RESULTS: In a review of 373 student responses (80% response rate), 255 students expressed an emotion-based reply, of which 93.75 were negative with such comments as they felt horrified, appalled, and disappointed by the patient's experience. Of the 227 students who commented on blame or fault, 70% felt that a system error was at fault, 45% felt that it was the provider and only 1.3% stated that it was the patient's fault. Of the students who mentioned the context or situation, just more than half felt that the episode was preventable and there was a causal relationship between clinic workers' actions and the medical error. Finally, a high percentage of students had a solutions-oriented response, noting the importance of teamwork and communication in the avoidance of medical errors. CONCLUSIONS: First-year health profession students responded to medical error with strong emotion and distress. Their responses largely represent an individualistic view of healthcare in both the causes and solutions to medical errors. No differences in response themes were observed by discipline. This study represents our students' emotional responses to a medical error scenario. The qualitative responses and reactions of students to the Sheridan video offered opportunities to tease out nuances that would otherwise be unavailable in a typical attitudes survey. We noted a individualistic view of healthcare in both the causes and solutions to medical errors. We view these results as an opportunity for interprofessional education in systems-level approaches to improve patient safety. Curricular efforts in interprofessional education, collaborative practice, and patient safety should be driven by these results.

Interprofessional simulation training's impact on process and outcome team efficacy beliefs over time.

Introduction: Recent findings suggest that process and outcome-based efficacy beliefs are factorially distinct with differential effects for team performance. This study extends this work by examining process and outcome efficacy (TPE, TOE) of interprofessional (IP) care teams over time. Methods: A within-team, repeated measures design with survey methodology was implemented in a sample of prelicensure IP care teams performing over three consecutive clinical simulation scenarios. TPE and TOE were assessed before and after each performance episode. Results: Initial baseline results replicated the discriminant validity for TPE and TOE separate factors. Further findings from multilevel modelling indicated significant time effects for TPE convergence, but not TOE convergence. However, a cross-level interaction effect of 'TOE(Start-Mean)×Time' strengthened TOE convergence over time. A final follow-up analysis of team agreement's substantive impact was conducted using independent faculty-observer ratings of teams' final simulation. Conclusion: Independent sample t-tests of high/low-agreement teams indicated support for agreement's substantive impact, such that high-agreement teams were rated as significantly better performers than low-agreement teams during the final simulation training. We discuss the substantive merit of methodological within-team agreement as an indicator of team functionality within IP and greater healthcare-simulation trainings at-large.

Mindfulness fostering of interprofessional simulation training for collaborative practice.

Introduction The setting demands imposed by performing in new, interdisciplinary cultures is common for modern healthcare workers. Both health science students and evidence-based workers are required to operate in professional cultures that differ from their own. As health organisations have placed increasing value on mindfulness for improving performance outcomes, so too have educational administrators embraced common, mindful competencies for improving training for improved patient outcomes. The training of future clinicians for diversified care. teams and patient populations has become known as interprofessional education (IPE). Although the goals for IPE suggest that individual differences in trait mindfulness may serve an important determinant for training effectiveness, it has gone unstudied in extant simulation training research. MethodsTo fill this gap, in this paper, we examine trait mindfulness' predictive power for training outcomes across two IPE cohort samples using two, prospective observational designs. Results Study 1's Findings supported trait mindfulness' prediction of perceived teamwork behaviours in training simulations between medical and nursing students (n=136). In study 2's expanded sample to five health professions (n=232), findings extended trait mindfulness' prediction of team efficacy and skill transfer, assessed 1 month after training. Conclusion A final, follow-up assessment 16 months later extended mindfulness' predictive validity to knowledge retention and teamwork attitudes. We discuss the theoretical and practical implication of our findings for advancing mindfulness research and IPE effectiveness assessment.

Early Impact of the Western Journal of Emergency Medicine CDEM/CORD Special Issue in Educational Research & Practice.

INTRODUCTION: In 2015, with a stated goal of disseminating best teaching practices and developing a community of educational scholars, the Council of Emergency Medicine Directors (CORD) and the Clerkship Directors of Emergency Medicine (CDEM) created an annual Special Issue in Educational Research and Practice (Special Issue) in cooperation with the Western Journal of Emergency Medicine. The intention of this study was to analyze the impact of this effort to date. METHODS: Bibliometric data was gathered on all four special issues, 2015-2019, from the Web of Science and then verified with the eScholarship website. Authorship, academic affiliation, date published, article type, and format were tabulated for descriptive analysis. Using metrics from Google Scholar, alternative scholarly impact metrics (altmetrics), and the eScholarship website, the authors identified top articles and grouped them into themes. RESULTS: Of the 136 articles included in the first four years of the Special Issue, 126 represented peer-reviewed publications with an overall acceptance rate of 25.0% (126/505). Authors from this cohort represented 103 of the 182 (56.6%) Accreditation Council for Graduate Medical Education (ACGME) programs in existence at the time of the inaugural issue. Multi-institutional studies represented 34.9% (44/126) of the peer-reviewed publications. Traditional and alternative publication metrics are reported to assess the impact of articles from the Special Issues. CONCLUSION: The Special Issue is a proven outlet to share best practices, innovations, and research related to education. Additionally, the infrastructure of this process promotes the development of individual faculty and a community of teaching scholars.

Implementing a third-year emergency medicine medical student curriculum.

BACKGROUND: Emergency medicine (EM) is commonly introduced in the fourth year of medical school because of a perceived need to have more experienced students in the complex and dynamic environment of the emergency department. However, there is no evidence supporting the optimal time or duration for an EM rotation, and a number of institutions offer third-year rotations. OBJECTIVE: A recently published syllabus provides areas of knowledge, skills, and attitudes that third-year EM rotation directors can use to develop curricula. This article expands on that syllabus by providing a comprehensive curricular guide for the third-year medical student rotation with a focus on implementation. DISCUSSION: Included are consensus-derived learning objectives, discussion of educational methods, considerations for implementation, and information on feedback and evaluation as proposed by the Clerkship Directors in Emergency Medicine Third-Year Curriculum Work Group. External validation results, derived from a survey of third-year rotation directors, are provided in the form of a content validity index for each content area. CONCLUSIONS: This consensus-derived curricular guide can be used by faculty who are developing or revising a third-year EM medical student rotation and provide guidance for implementing this curriculum at their institution.

Comparison of expert and novice sonographers' performance in focused lung ultrasonography in dyspnea (FLUID) to diagnose patients with acute heart failure syndrome.

OBJECTIVES: The goal of this study was to examine the ability of emergency physicians who are not experts in emergency ultrasound (US) to perform lung ultrasonography and to identify B-lines. The hypothesis was that novice sonographers are able to perform lung US and identify B-lines after a brief intervention. In addition, the authors examined the diagnostic accuracy of B-lines in undifferentiated dyspneic patients for the diagnosis of acute heart failure syndrome (AHFS), using an eight-lung-zone technique as well as an abbreviated two-lung-zone technique. METHODS: This was a prospective, cross-sectional study of patients who presented to the emergency department (ED) with acute dyspnea from May 2009 to June 2010. Emergency medicine (EM) resident physicians, who received a 30-minute training course in thoracic US examinations, performed lung ultrasonography on patients presenting to the ED with undifferentiated dyspnea. They attempted to identify the presence or absence of sonographic B-lines in eight lung fields based on their bedside US examinations. An emergency US expert blinded to the diagnosis and patient presentation, as well as to the residents' interpretations of presence of B-lines, served as the criterion standard. A secondary outcome determined the accuracy of B-lines, using both an eight-lung-zone and a two-lung-zone technique, for predicting pulmonary edema from AHFS in patients presenting with undifferentiated dyspnea. Two expert reviewers who were blinded to the US results determined the clinical diagnosis of AHFS. RESULTS: A cohort of 66 EM resident physicians performed lung US on 380 patients with a range of 1 to 28 examinations, a mean of 5.8 examinations, and a median of three examinations performed per resident. Compared to expert interpretation, lung US to detect B-lines by inexperienced sonographers achieved the following test characteristics: sensitivity 85%, specificity 84%, positive likelihood ratio (+LR) 5.2, negative likelihood ratio (-LR) 0.2, positive predictive value (PPV) 64%, and negative predictive value (NPV) 94%. Regarding the secondary outcome, the final diagnosis was AHFS in 35% of patients (134 of 380). For novice sonographers, one positive lung zone (i.e., anything positive) had a sensitivity of 87%, a specificity of 49%, a +LR of 1.7, a -LR of 0.3, a PPV of 50%, and an NPV of 88% for predicting AHFS. When all eight lung zones were determined positive (i.e., totally positive) by novice sonographers, the sensitivity was 19%, specificity was 97%, +LR was 5.7, -LR was 0.8, PPV was 76%, and NPV was 68% for predicting AHFS. The areas under the curve for novice and expert sonographers were 0.77 (95% CI = 0.72 to 0.82) and 0.76 (95% CI = 0.71 to 0.82), respectively. CONCLUSIONS: Novice sonographers can identify sonographic B-lines with similar accuracy compared to an expert sonographer. Lung US has fair predictive value for pulmonary edema from acute heart failure in the hands of both novice and expert sonographers.

Teaching the teachers: faculty development in inter-professional education.

AIMS: The purpose of this study is to evaluate changes in self-concept for the knowledge, skills and attitudes toward inter-professional teamwork of facilitators who participated in training and an inter-professional team training event. BACKGROUND: Inter-professional education requires dedicated and educated faculty. METHODS: A pretest posttest quasi-experimental design was used for the evaluation. Fifty-three facilitators were asked to complete pre-post questionnaires to measure inter-professional team self-concept (IPTSC), assessing self-concept for the knowledge, skills, and attitudes required for performing in an inter-professional team. RESULTS: Post-session scores on inter-professional team knowledge, skills and attitudes were significantly higher (F(1, 31) = 5.59, p = .02). CONCLUSION: A facilitator development course and participation in the teaching event had a positive impact on perceived knowledge, skills and attitudes toward inter-professional teamwork.

Evaluating Simulation-Based ACLS Education on Patient Outcomes: A Randomized, Controlled Pilot Study.

BACKGROUND: Simulation training is widely accepted as an effective teaching tool, especially for dealing with high-risk situations. OBJECTIVE: We assessed whether standardized, simulation-based advanced cardiac life support (ACLS) training improved performance in managing simulated and actual cardiac arrests. METHODS: A total of 103 second- and third-year internal medicine residents were randomized to 2 groups. The first group underwent conventional ACLS training. The second group underwent two 2 1/2-hour sessions of standardized simulation ACLS teaching. The groups were assessed by evaluators blinded to their assignment during in-hospital monthly mock codes and actual inpatient code sheets at 3 large academic hospitals. Primary outcomes were time to initiation of cardiopulmonary resuscitation, time to administration of first epinephrine/vasopressin, time to delivery of first defibrillation, and adherence to American Heart Association guidelines. RESULTS: There were no differences in primary outcomes among the study arms and hospital sites. During 21 mock codes, the most common error was misidentification of the initial rhythm (67% [6 of 9] and 58% [7 of 12] control and simulation arms, respectively, P  =  .70). There were no differences in primary outcome among groups in 147 actual inpatient codes. CONCLUSIONS: This blinded, randomized study found no effect on primary outcomes. A notable finding was the percentage of internal medicine residents who misidentified cardiac arrest rhythms.

Emergency medicine clerkship directors: current workforce.

INTRODUCTION: The emergency medicine clerkship director serves an important role in the education of medical students. The authors sought to update the demographic and academic profile of the emergency medicine clerkship director. METHODS: We developed and implemented a comprehensive questionnaire, and used it to survey all emergency medicine clerkship directors at United States allopathic medical schools accredited by the Liaison Committee on Medical Education. We analyzed and interpreted data using descriptive statistics. RESULTS: One hundred seven of 133 (80.4%) emergency medicine clerkship directors completed the survey. Clerkship Director's mean age was 39.7 years (SD-7.2), they were more commonly male 68.2%, of Caucasian racial backgrounds and at the instructor or assistant professor (71.3%) level. The mean number of years of experience as clerkship director was 5.5 (SD-4.5). The mean amount of protected time for clerkship administration reported by respondents was 7.3 hours weekly (SD-5.1), with the majority (53.8%) reporting 6 or more hours of protected time per week. However, 32.7% of emergency medicine clerkship directors reported not having any protected time for clerkship administration. Most clerkship directors (91.6%) held additional teaching responsibilities beyond their clerkship and many were involved in educational research (49.5%). The majority (79.8%), reported being somewhat or very satisfied with their job as clerkship director. CONCLUSION: Most clerkship directors were junior faculty at the instructor or assistant professor rank and were involved with a variety of educational endeavors beyond the clerkship.

State of undergraduate education in emergency medicine: a national survey of clerkship directors.

BACKGROUND: The discipline of emergency medicine (EM) has rapidly changed over the past 10 years, resulting in greater involvement of the specialty in undergraduate medical education. OBJECTIVES: The authors sought to present a review of how, when, and where EM is currently taught in U.S. medical schools and to include general program characteristics, such as number of required clerkships, clinical expectations and experiences, use of the national curriculum guide, didactic content, and methods of assessment and grading. METHODS: The authors surveyed representatives of the 128 U.S. allopathic medical schools on medical education in 2010. Contacts were drawn from established databases, direct inquiries, and medical school websites. Up to five attempts were made to contact representatives through e-mail and telephone. Descriptive statistics were used to summarize the data. RESULTS: The survey response rate was 83.6%. Fifty-two percent of medical schools now require students to complete EM clerkships. Required EM clerkships usually last 4 weeks and take place during the fourth year of medical school. They require students to complete a mean (±SD) of 14.3 (±2.8) shifts, which average 8.9 (±1.4) hours in length. Programs include a mean (±SD) of 18 (±10.4) hours of didactics. Approximately 60% of respondents report that both residents and attending physicians precept students. Assessments of students primarily include written clinical performance assessments and end-of-rotation written tests. These assessments contribute 66.8 and 24.5%, respectively, to the clerkship grade. CONCLUSIONS: Currently more than half of all U.S. medical schools require EM clerkships in their undergraduate medical curricula. This article reports an overview of EM programs at the undergraduate level.

Reporter-Interpreter-Manager-Educator (RIME) descriptive ratings as an evaluation tool in an emergency medicine clerkship.

BACKGROUND: Emergency Medicine (EM) clerkships traditionally assess students using numerical ratings of clinical performance. The descriptive ratings of the Reporter, Interpreter, Manager, and Educator (RIME) method have been shown to be valuable in other specialties. OBJECTIVES: We hypothesized that the RIME descriptive ratings would correlate with clinical performance and examination scores in an EM clerkship, indicating that the RIME ratings are a valid measure of performance. METHODS: This was a prospective cohort study of an evaluation instrument for 4(th)-year medical students completing an EM rotation. This study received exempt Institutional Review Board status. EM faculty and residents completed shift evaluation forms including both numerical and RIME ratings. Students completed a final examination. Mean scores for RIME and clinical evaluations were calculated. Linear regression models were used to determine whether RIME ratings predicted clinical evaluation scores or final examination scores. RESULTS: Four hundred thirty-nine students who completed the EM clerkship were enrolled in the study. After excluding items with missing data, there were 2086 evaluation forms (based on 289 students) available for analysis. There was a clear positive relationship between RIME category and clinical evaluation score (r(2)=0.40, p<0.01). RIME ratings correlated most strongly with patient management skills and least strongly with humanistic qualities. A very weak correlation was seen with RIME and final examination. CONCLUSION: We found a positive association between RIME and clinical evaluation scores, suggesting that RIME is a valid clinical evaluation instrument. RIME descriptive ratings can be incorporated into EM evaluation instruments and provides useful data related to patient management skills.

Emergency medicine clerkship curriculum: an update and revision.

In 2006, the latest version of a national curriculum for the fourth-year emergency medicine (EM) clerkship was published. Over the past several years, that curriculum has been implemented across multiple clerkships. The previous curriculum was found to be too long and detailed to cover in 4 weeks. As well, updates to the Liaison Committee on Medical Education (LCME)'s form and function document, which guides the structure of a clerkship, have occurred. Combining experience, updated guidelines, and the collective wisdom of members of the national organization of the Clerkship Directors in Emergency Medicine (CDEM), an update and revision of the fourth-year EM clerkship educational syllabi has been developed.

The use of simulation and a modified TeamSTEPPS curriculum for medical and nursing student team training.

INTRODUCTION: We describe our adaptation of the team strategies and tools to enhance performance and patient safety (TeamSTEPPS) for use as an educational intervention for medical and nursing students. We hypothesized that participation in a team training program using the modified TeamSTEPPS program would positively affect knowledge and attitudes toward teamwork skills and their ability to recognize the presence and quality of team skills. METHODS: Two hundred thirteen students participated in a 4-hour team training program that included a lecture followed by small group team training exercises. Nurse/physician pairs facilitated student team activities. Knowledge and attitudes were assessed before and after the educational intervention. Recognition of team skills was assessed using videos. RESULTS: Statistically significant differences were found with participant knowledge (P<0.001) and attitude (P=0.004). Students were able to identify the presence and quality of team skills in the video vignettes. Overwhelmingly, the students recognized team skills much better in the success video than in the opportunity video (P<0.001), and they rated the success video much higher than the opportunity video (P<0.001) with regards to the quality of team skills. CONCLUSIONS: We successfully adapted TeamSTEPPS for use with our medical and nursing students, educated facilitators, and implemented the course. Our students improved their knowledge of vital team and communication skills, attitudes toward working as teams, and were able to identify effective team skills.

Standardized Patients to Teach Medical Students about Intimate Partner Violence.

OBJECTIVE: To use 360-degree evaluations within an Observed Structured Clinical Examination (OSCE) to assess medical student comfort level and communication skills with intimate partner violence (IPV) patients. METHODS: We assessed a cohort of fourth year medical students' performance using an IPV standardized patient (SP) encounter in an OSCE. Blinded pre- and post-tests determined the students' knowledge and comfort level with core IPV assessment. Students, SPs and investigators completed a 360-degree evaluation that focused on each student's communication and competency skills. We computed frequencies, means and correlations. RESULTS: Forty-one students participated in the SP exercise during three separate evaluation periods. Results noted insignificant increase in students' comfort level pre-test (2.7) and post-test (2.9). Although 88% of students screened for IPV and 98% asked about the injury, only 39% asked about verbal abuse, 17% asked if the patient had a safety plan, and 13% communicated to the patient that IPV is illegal. Using Likert scoring on the competency and overall evaluation (1, very poor and 5, very good), the mean score for each evaluator was 4.1 (competency) and 3.7 (overall). The correlations between trainee comfort level and the specific competencies of patient care, communication skill and professionalism were positive and significant (p<0.05). CONCLUSION: Students felt somewhat comfortable caring for patients with IPV. OSCEs with SPs can be used to assess student competencies in caring for patients with IPV.